Object sensing apparatus

ABSTRACT

An object sensing apparatus for driver assistance systems in motor vehicles, including at least two sensor systems which measure data concerning the location and/or motion status of objects in the vicinity of the vehicle, and whose detection regions overlap one another, characterized by an error recognition device that checks the data measured by the sensor systems for absence of contradictions, and outputs an error signal upon detection of a contradiction.

FIELD OF THE INVENTION

The present invention relates to an object sensing apparatus for driverassistance systems in motor vehicles, including at least two sensorsystems which measure data concerning the location and/or motion statusof objects in the vicinity of the vehicle, and whose detection regionsoverlap one another.

BACKGROUND INFORMATION

Motor vehicles are increasingly being equipped with driver assistancesystems that assist and provide support to the driver in driving thevehicle. One example of such an assistance system is a so-calledadaptive cruise control (ACC) system, which automatically regulates thevehicle's speed to a desired speed selected by the driver or, if apreceding vehicle is present, adapts the speed in such a manner that asuitable distance from the preceding vehicle, monitored with the aid ofa distance sensor, is maintained. Other examples of driver assistancesystems are collision warning devices; automatic lane keeping systems(LKS), which detect roadway markings and automatically keep the vehiclein the center of the lane by intervening in the steering system;sensor-assisted parking aids, and the like. All these assistance systemsrequire a sensor system with which information concerning the vehicle'svicinity may be sensed, as well as evaluation units with which thatinformation may be suitably evaluated and interpreted.

These devices are capable of detecting objects in the vehicle'svicinity, for example other vehicles and additional obstacles, andsensing data that characterize the location and, if applicable, themotion status of those objects. The sensor systems and associatedevaluation units will therefore be referred to in combination as anobject sensing apparatus.

Examples of sensor systems that are used in such object sensingapparatuses are radar systems and their optical counterparts (so-calledlidar systems), as well as stereo camera systems. With radar systems,the distance of the object along the line of sight may be measured byevaluating the transit time of the radar echo. The relative velocity ofthe object along the line of sight may also be measured directly byevaluating the Doppler shift of the radar echo. With adirection-sensitive radar system, for example a multi-beam radar, it isalso possible to sense directional data concerning objects, for examplethe azimuth angle relative to a reference axis defined by the alignmentof the radar sensor. With stereo camera systems, directional data andalso (by parallax evaluation) distance data may be obtained. Byevaluating the raw data measured directly by these sensor systems, it ispossible to calculate data that indicate the distance of the object inthe direction of travel, as well as the transverse offset of the objectrelative to the center of the roadway or relative to the instantaneousstraight-ahead orientation of the vehicle.

Since conventional sensor systems have their strengths and weaknesses asregards sensing of the requisite measured data, it is advisable to useseveral sensor systems that supplement one another.

In ACC systems, it is conventional to subject the measured raw data to aplausibility evaluation in order to decide, or at least to indicateprobabilities, as to whether the object sensed is a relevant obstacle oran irrelevant object, for example a sign at the side of the road. Insome circumstances, an implausibility in the sensed data may alsoindicate a defect in the sensor system.

It general, however, it is not possible with conventional object sensingapparatuses reliably to detect misalignments or other defects in thesensor systems that negatively affect the functionality of theassistance system.

SUMMARY

According to an example embodiment of the present invention, an objectsensing apparatus is provided with which it may be possible to detectdefects in the sensor systems during operation more accurately and morereliably, and thus to improve the functional dependability of anassistance system.

According to an example embodiment of the present invention, an errorrecognition device checks the data measured by the sensor systems forabsence of contradictions, and outputs an error signal upon detection ofa contradiction.

As aspect of an example embodiment of the present invention is based onthe consideration that when several sensor systems including mutuallyoverlapping detection regions are present, it may often be the case thatobjects are located in the overlap region. In this situation, the sensorsystems operating independently of one another furnish redundantinformation that makes possible error detection while the apparatus isin operation. When the participating sensor systems are operatingcorrectly, the data furnished by them may be compatible with one anotherwithin certain error limits. If that is not the case—i.e. if the datacontradict one another—it may be deduced therefrom that at least one ofthe participating sensor systems is defective, and an error signal isoutputted. In one case, this error signal may be used to inform thedriver of the malfunction by an optical or acoustic indicator and, ifapplicable, to initiate automatic deactivation of the assistance system.According to an example embodiment of the present invention, anautomatic error correction may be performed using this error signal.

An example embodiment of the present invention thus may make possible acontinuous self-diagnosis of the object sensing apparatus during normalvehicle operation, and thus a substantial improvement in the drivingsafety of the assistance system that uses the data of the object sensingapparatus.

In an example embodiment, the object sensing apparatus includes, inaddition to a sensor system for the long-range region that isconstituted e.g. by a 77-GHz radar system or a lidar system, a sensorsystem for the short-range region that has a shorter reach but also alarger angular region, so that dead angles in the short-range region maybe largely eliminated. The sensor system for the short-range region mayinclude a radar system or by a lidar system, or also by a video sensorsystem, for example a stereo camera system including two electroniccameras.

In an example embodiment, three mutually independent sensor systemswhose detection regions include a shared overlap region may be present.There exists in this case, for objects that are located in the sharedoverlap region, a capability not only for error detection but also foreasily identifying the faulty sensor system by “majority decision,” andoptionally for correcting the data, the alignment, or the calibration ofthe faulty system.

Automatic identification of the faulty system and automatic errorcorrection are possible in specific circumstances even in exampleembodiments including only two sensor systems, e.g., by a plausibilityevaluation in consideration of the details of the physical measurementprinciples used in the participating sensor systems. For example, arelatively accurate distance measurement is possible with radar andlidar systems, whereas distance measurement using a stereo camera systemmay involve greater error tolerances (especially at longer distances)and may depend critically on camera alignment. In the event of adiscrepancy, therefore, a fault in the stereo camera system is highlyprobable. Conversely, a video system may permit a relatively accuratemeasurement of the transverse offset of a preceding vehicle, whereastransverse offset measurement by a radar or lidar system may dependcritically on the alignment of the radar or lidar system. In this case,therefore, a discrepancy is more suggestive of a defect in the radar orlidar system.

In practice, the region in which the detection regions of the sensorsystems overlap will be a region that is of relevance for the assistancesystem. In an ACC system, for example, the sensor systems for theshort-range region and the long-range region may be configured so thatthey overlap in the distance region that corresponds to the typicalsafety distance from a preceding vehicle. In this case automatic errorcorrection and an improvement in measurement accuracy may also beachieved by weighting the data furnished by the various sensor systemsin accordance with their respective reliability, and then combining themto yield a final result.

According to an example embodiment of the present invention, it may beprovided to store the error signals furnished by the error recognitiondevice together with the associated mutually contradictory measurementdata, and thus to create error statistics that facilitate diagnosis whenthe object sensing apparatus is repaired or maintained.

Example embodiments of the present invention will be explained in moredetail below with reference to the appended Figures.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 schematically illustrates the detection regions of several sensorsystems that are installed on a motor vehicle.

FIG. 2 is a block diagram of an object sensing apparatus according to anexample embodiment of the present invention.

FIG. 3 illustrates the consequences of misalignments of various sensorsystems.

DETAILED DESCRIPTION

FIG. 1 illustrates, in a schematic plan view, the front part of a motorvehicle 10 that is equipped with three sensor systems operatingindependently of one another, namely a long-range radar 12, ashort-range radar 14, and a video system that is constituted by twocameras 16L and 16R. Long-range radar 12 includes a detection region 18having a range of, for example 150 m and a sensing angle of 15°, whileshort-range radar 14 includes a detection region 20 having a range of,for example, 50 m and a sensing angle of 40°. Between these detectionregions 18, 20, which are not illustrated to scale in the drawings,there exists an overlap region 22. The detection region of the videosystem constituted by cameras 16L, 16R, however, which together will belabeled with the reference character 16, includes overlap region 22(when visibility conditions are good). An object 24 that is located inthis overlap region 22 may therefore be sensed by all three sensorsystems.

Detection region 18 of long-range radar 12 is symmetrical with respectto a reference axis 18A that, when the radar sensor is correctlyaligned, extends parallel to a main axis H which extends in thelongitudinal direction through the center of vehicle 10. Detectionregion 20 of short-range radar 14 is accordingly symmetrical withrespect to a reference axis 20A that is parallel to main axis H and toreference axis 18A.

Long-range radar 12 measures distance d1 to object 24 as well as therelative velocity of object 24 relative to vehicle 10, and azimuth anglej1 of object 24 relative to reference axis 18A. Close-range radar 14correspondingly measures distance d2 to object 24, the relative velocityof object 24 along the line of sight from the radar sensor to theobject, and azimuth angle j2 of object 24 relative to reference axis20A.

The images of object 24 acquired by cameras 16L, 16R are evaluatedelectronically in video system 16. The evaluation software of suchstereo camera systems is able to identify object 24 in the imagesacquired by the two cameras and to determine, based on the parallaxshift, the location of object 24 in a two-dimensional coordinate system(parallel to the road surface). In this fashion, video system 16furnishes perpendicular distance d3 of object 24 from vehicle 10 (i.e.from the baseline of cameras 16L, 16R), and transverse offset y3 ofobject 24 with respect to main axis H.

The local coordinates of object 24 may thus be determined in threemutually independent manners using the three sensor systems 12, 14, 16.The polar coordinates measured by the radar systems may be converted bya coordinate transformation into Cartesian coordinates, as constitutedby the coordinate pair (d3, y3) in the example illustrated. The threecoordinate sets measured independently of each other may then becompared to one another; if these coordinates contradict one another,this indicates that one of the three sensor systems is operatingdefectively. The faulty system may also be identified on the basis ofthe discrepant coordinate set.

The relative velocity of the object 24 may also be determined bydifferentiation over time of distance d3 measured with video system 16.Since the lines of sight from the radar sensors to object 24, alongwhich the relative velocities are measured using the Doppler effect, arenot exactly parallel to main axis H, the three measured relativevelocities will differ slightly from one another. Under the distanceconditions that occur in practice, however, this discrepancy may benegligible. If necessary, it may be corrected by conversion intoCartesian coordinates, so that the measured velocity data may also becompared with and adjusted to one another.

FIG. 2 is a block diagram of an object sensing apparatus thatencompasses long-range radar 12, short-range radar 14, video system 16,and associated evaluation units 26, 28, 30, and furthermore an errorrecognition device 32 and a correction device 34. Evaluation devices 26,28, 30, error recognition device 32, and correction device 34 may beconstituted by electronic circuits, by microcomputers, or also bysoftware modules in a single microcomputer.

From the raw data furnished by long-range radar 12, evaluation unit 26determines distances d1 i, relative velocities v1 i, and azimuth anglesj1 i of all objects that are present in sensing region 18 of long-rangeradar 12. Index i serves here to identify the individual objects. Fromthe distance data and azimuth angles, evaluation unit 26 also calculatestransverse offsets y1 i of the various objects.

In similar fashion, evaluation unit 28 determines distances d2 i,relative velocity v2 i, azimuth angles j2 i, and transverse offsets y2 iof all objects that are present in sensing region 20 of short-rangeradar 14.

Evaluation unit 30 firstly determines azimuth angles jLi and jRi of theobjects sensed by cameras 16L, 16R. These azimuth angles are definedanalogously to azimuth angles j1 and j2 in FIG. 1, i.e. they indicatethe angle between the respective line of sight to the object and astraight line parallel to main axis H. Distances d3 i, transverseoffsets y3 i, and (by differentiation of the distance data over time)relative velocities v3 i are calculated on the basis of azimuth anglesjLi and jRi and the known distance between cameras 16L and 16R.

Distances d1 i, d2 i, and d3 i determined by the three evaluation units26, 28, 30 are conveyed to a distance module 36 of error recognitiondevice 32. Correspondingly, relative velocity data v1 i, v2 i, and v3 iare conveyed to a velocity module 38, and transverse offset data y1 i,y2 i, and y3 i to a transverse offset module 40. An angle module 42 oferror recognition device 32 evaluates azimuth angles j1 i, j2 i, jLi,and jRi.

The various modules of error recognition device 32 are connected to oneanother, and have access to all the data that are conveyed to errorrecognition device 32 from any of the evaluation units. The dataconnections illustrated in the drawings refer in each case only to thedata processed on a foreground basis in the relevant module.

When evaluation unit 26 reports distance di1 of a sensed object (havingindex i) to distance module 36, distance module 36 then first checks, onthe basis of the associated transverse offset y1 i, whether the objectin question is also located in sensing region 20 of short-range radar 14and/or in the sensing region of video system 16. If that is the case,the distance module checks whether data for that object are alsoavailable from evaluation units 28, 30. Identification of the objects isfacilitated by the fact that distance module 36 may track the changeover time in the distance data. For example, if an object is initiallysensed only by long-range radar 12 and then enters sensing region 20 ofshort-range radar 14, it may be expected that evaluation unit 28 willreport the occurrence of a new object that may then be identified withthe tracked object. To eliminate ambiguities, it is also possible toemploy the criterion that the local coordinates transmitted by thevarious evaluation units for the same object are at least approximatelyconsistent with one another.

If distance data are available for a single object from several sensorsystems, distance module 36 checks whether those distance data areconsistent within the respective error limits. It is considered in thiscontext that the error limits are themselves variable. For example,transverse offset data y1 i are relatively inaccurate for a large objectdistance, since long-range radar 12 has only a limited angularresolution and even slight deviations in the measured azimuth angleresult in a considerable deviation in the associated transverse offset.If the distance data are consistent within the error limits, theconsistent value di is transmitted to a downstream assistance system 44,for example an ACC system. The outputted value di may be a weightedaverage of distance data d1 i, d2 i, and d3 i, the weights being greaterin proportion to the reliability of the data of the sensor system inquestion.

Distance data d2 i and d3 i transmitted from evaluation units 28 and 30are evaluated by distance module 36 in a manner corresponding to thatfor data d1i from evaluation unit 26. Thus, for example, if an object isinitially sensed only by short-range radar 14 and then migrates intosensing region 18 of long-range radar 12, distance module 36 thusinitially tracks the change in the data arriving from evaluation unit 28and then checks whether corresponding data also arrive, at the expectedpoint in time, from evaluation unit 26.

If the expected data from one of evaluation units 26, 28, 30 are absent,i.e. if a sensor system does not sense an object even though that objectshould, to judge by the data from the other systems, be located in thesensing region, distance module 36 then outputs an error signal Fdj.Index j here identifies the sensor system from which no data wereobtained. Error signal Fdj thus indicates that the sensor system inquestion has possibly failed or is “blind.”

If distance module 36 contains all the expected distance data but ifthose data deviate from one another by more than the error limits, errorsignal Fdj is once again outputted. In this case, error signal Fdj alsoindicates the sensor systems from which the discrepant data wereobtained, and the magnitude of the discrepancy.

If distance data that are consistent within the error limits areavailable from at least two sensor systems, distance value di may becreated from those data and outputted to assistance system 44 eventhough an error has been identified and error signal Fdj has beengenerated.

The manner of operation of velocity module 38 is analogous to the mannerof operation (described above) of distance module 36, except that hereit is not the distance data but rather velocity data v1 i, v2 i, and v3i that are compared with one another, in order to create therefrom avelocity value vi that is outputted to assistance system 44, and/or tooutput an error signal Fvj that indicates a discrepancy between themeasured relative velocities.

The manner of operation of transverse offset module 40 is also largelyanalogous to the manner of operation of distance module 36 and velocitymodule 38 as described above. In the example illustrated, however, noprovision is made for output of an error signal in this case, since thetransverse offset data are merely derived data that are calculated fromthe measured azimuth angles, so that the azimuth angle should beprimarily relied upon for error recognition.

Azimuth angles j1 i, j2 i, jLi, and jRi are accordingly comparedseparately in angle module 42. In the comparison of these azimuthangles, consideration is given to the discrepancies that necessarilyresult, for a given object, from the object distance and the variouspositions of the relevant sensors or cameras on the baseline. If adiscrepancy exceeding the error limits remains when these deviationshave been considered, an error signal Ffk is outputted. Index k (k=1through 4) in this case identifies camera 16L or 16R or the radar sensorwhose azimuth angle or angles do not match the other azimuth angles. Ifa sufficiently reliable determination of the transverse offset ispossible despite the discrepancy that has been identified, acorresponding value yi for the transverse offset is outputted fromtransverse offset module 40 to assistance system 44.

In the example illustrated, error signals Fdj, Fvj, and Ffk are conveyedto correction device 34. If the error signals indicate with sufficientcertainty which of the three sensor systems is responsible for thediscrepancy, and if it is evident from the nature and magnitude of theidentified error that the error may be corrected by a recalibration ofthe sensor system in question, a correction signal K is then outputtedto the associated evaluation unit 26, 28, or 30. Optionally, thecorrection signal may also be outputted directly to long-range radar 12,short-range radar 14, or video system 16.

One example of a systematic error that may be corrected by recalibrationis a misalignment of a radar sensor or a camera, which causes adeflection of the reference axis in question (e.g. 18A or 20A) and thusan incorrect measurement of the azimuth angle. In this case thecalibration may be modified in the relevant evaluation unit in such amanner that the misalignment is corrected and the correct azimuth angleis once again obtained. Although the misalignment should still beremedied at the next service (since the misalignment also results in anundesirable change in the sensing region), system functionality maynevertheless be temporarily maintained by recalibrating.

In the example embodiment illustrated, correction device 34 includes astatistics module 46 which stores error signals Fdj, Fvj, and Fjk thathave occurred during operation of the apparatus, and thus documents thenature and magnitude of all the errors that occur. These data are thenavailable for diagnostic purposes when the apparatus is serviced orrepaired. Statistics module 46 additionally has, in the exampleillustrated, the function of deciding whether an error may beautomatically corrected or whether an irresolvable error is present andan optical or acoustic error message F is outputted to the driver inorder to inform him or her of the malfunction. Error message F isoutputted, for example, when the signals obtained from error correctiondevice 28 indicate a total failure of one of the sensor systems. Thefunctions of statistics module 46 offer the possibility of notimmediately outputting error message F in the case of a discrepancy thatoccurs only once or sporadically, but outputting the error message onlywhen discrepancies of the same kind occur with a certain frequency. Therobustness of the apparatus may thereby be considerably improved.

FIG. 3 illustrates, using examples, the effect of a sensor misalignmenton the measurement result.

If main axis 18A is deflected through angle Dj, for example as a resultof a misalignment of long-range radar 12, the measured azimuth angle j1is too great by an amount equal to that angle, and long-range radar 12“sees” object 24 not in its actual position, but in position 24′ drawnwith dashed lines. This results in an error Dy1 in the measuredtransverse offset. The farther away object 24 is from vehicle 10, thegreater this error.

If, on the other hand, left camera 16L of the video system has amisalignment of the same magnitude, the associated azimuth angle jL isthen distorted by an amount equal to the same angular deviation Dj, asindicated in FIG. 3 by a line of sight S drawn as a dot-dash line. Videosystem 16 then sees object 24 at the intersection of the lines of sightof the two cameras, i.e. in position 24″. It is evident that in thiscase the error Dy3 measured for the transverse offset is substantiallysmaller. On the other hand, however, the misalignment of camera 16Lresults in a considerable error Dd3 in the distance measurement.

These relationships may be utilized in the apparatus described forautomatic error correction, even when only two sensor systems arepresent.

For example, if a misalignment of long-range radar 12 exists, acomparison of the transverse offset data from long-range radar 12 andfrom video system 16 yields a definite discrepancy Dy1, whereas thedistance data measured with the same systems are substantiallyconsistent. From this it may be concluded that the error is attributableto a misalignment of the radar system and not to a misalignment of acamera. It is even possible to determine the misalignment quantitativelyon the basis of the measured magnitude of the error, and to correct itby recalibration of the radar sensor or the associated evaluation unit26.

If, on the other hand, a misalignment of camera 16L exists, this isevident from a large discrepancy Dd3 in the distance data while thetransverse offset data are largely consistent. In this case the errormay be corrected by recalibrating the video system.

In the event of contradictory measurement results, other criteria mayalso be employed for the decision as to which of the participatingsensor systems is defective, e.g., including cases in which data fromonly two sensor systems are available for the object, or in which onlytwo sensor systems are in fact present on the vehicle.

When the transverse offset of object 24 with respect to reference axis18A is not zero, for example as in FIG. 3, azimuth angle j1 isapproximately inversely proportional to the object distance. In thiscase the rate of change of azimuth angle j1 is therefore dependent onthe relative velocity of the object, which is being directly measuredusing the radar system. If object 24 is in fact located on main axis18A, however, and if the transverse offset is merely being simulated bya misalignment Dj of the sensor system, the measured (apparent) azimuthangle is then independent of the distance and the relative velocity.Correspondingly, even when a transverse offset of the object actuallyexists, there is a discrepancy between the measured rate of change inthe azimuth angle and the rate of change predicted theoretically basedon the relative velocity. From this discrepancy, a misalignment of thesensor system may be deduced.

With video system 16, the possibility additionally exists of measuringthe distance-dependent change in the apparent size of object 24. Thisapparent change in size is directly proportional to the relativevelocity, and approximately inversely proportional to the distance. Anerror in the distance measurement caused by misalignment of a camera maythen be detected by the fact that the apparent change in size does notmatch the measured change in distance.

Since the nature of the object (e.g. a passenger car or a truck) mayalso be recognized using camera system 16, and since the typical actualsize of such objects is at least approximately known, it is alsopossible to check whether the object distance measured with camerasystem 16 is compatible with the measured apparent size of the object.

If road markings are also detected using camera system 16, thisinformation may also be utilized for automatic error recognition anderror correction. The transverse offset of the vehicle itself relativeto the center of the road may be detected based on the detected roadmarkings. It is to be expected that as a statistical average, thistransverse offset has a value of zero. If the evaluation in statisticsmodule 46 shows that the measured transverse position of the vehicleitself continuously deviates in one direction away from the center ofthe lane, this indicates misalignment of one or both cameras. This isapplicable only if the transverse offset of an object measured with thecamera system deviates in the same direction from the transverse offsetmeasured with another sensor system.

1. An object sensing device for a driver assistance system in a motorvehicle, comprising: at least two sensor systems which measure dataconcerning at least one of a location status and a motion status of anobject in a vicinity of the motor vehicle, detection regions of thesensor systems overlap one another; and an error recognition arrangementconfigured to check the data for absence of contradictions and to outputan error signal upon detection of a contradiction; wherein the errorrecognition arrangement checks, on a basis of the data, whether theobject is located in an overlap region between a first detection regionof a first sensor system and a second detection region of a secondsensor system, and outputs the error signal one of when the object isnot located by the second sensor system and when data measured by thesecond sensor system, taking into account error limits, deviate fromdata measured by the first sensor system; and wherein for objects thatare located in the overlap region, the error recognition device weightsthe data measured by the first sensor system and the second sensorsystem in accordance with reliability, and combines the weightedmutually corresponding data into data that are output to the driverassistance system.
 2. The object sensing device of claim 1, wherein thesensor systems include at least one of a radar system, a lidar system,and a video system.
 3. The object sensing device of claim 1, wherein thefirst sensor system is configured for a long-range region and the secondsensor system is configured for a short-range region.
 4. The objectsensing device of claim 3, wherein the first sensor system includes oneof a radar system and a lidar system.
 5. The object sensing device ofclaim 3, wherein the second sensor system includes one of a radarsystem, a lidar system, and a video system.
 6. The object sensing deviceof claim 1, wherein one of the sensor systems includes a video systemincluding at least two cameras for acquiring a stereo image of theobject.
 7. The object sensing device of claim 1, wherein the errorrecognition arrangement is configured to identify, on a basis of a typeof contradiction ascertained, which sensor system is causing the error.8. The object sensing device of claim 7, further comprising: acorrection arrangement configured to correct an ascertained error by oneof realigning and recalibrating one of the sensor system that is causingthe error and an evaluation unit belonging to that sensor system.
 9. Theobject sensing device of claim 7, further comprising: a statisticsmodule configured to record and store ascertained errors.
 10. The objectsensing device of claim 9, wherein the statistics module is configuredto output an error message to a driver when an error ascertained by theerror recognition arrangement occurs with a certain statisticalfrequency.